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Abstract 

Background: Electronic Case Report Forms (eCRFs) are increasingly chosen by investigators and sponsors of clinical 
research instead of the traditional pen-and-paper data collection (pCRFs). Previous studies suggested that eCRFs 
avoided mistakes, shortened the duration of clinical studies and reduced data collection costs. 

Methods: Our objectives were to describe and contrast both objective and subjective efficiency of pCRF and eCRF 
use in clinical studies. A total of 27 studies (1 1 eCRF, 16 pCRF) sponsored by the Paris hospital consortium, 
conducted and completed between 2001 and 201 1 were included. Questionnaires were emailed to investigators of 
those studies, as well as clinical research associates and data managers working in Paris hospitals, soliciting their 
level of satisfaction and preferences for eCRFs and pCRFs. Mean costs and timeframes were compared using 
bootstrap methods, linear and logistic regression. 

Results: The total cost per patient was 374€ ±351 with eCRFs vs. 1,1 35€ ±1,234 with pCRFs. Time between the 
opening of the first center and the database lock was 31.7 months Q1 = 24.6; Q3 = 42.8 using eCRFs, vs. 39.8 months 
Q1 =31.7; Q3 = 52.2 with pCRFs (p = 0.1 1). Electronic CRFs were globally preferred by all (31/72 vs. 15/72 for paper) for 
easier monitoring and improved data quality. 

Conclusions: This study found that eCRFs and pCRFs are used in studies with different patient numbers, center 
numbers and risk. The first ones are more advantageous in large, low-risk studies and gain support from a majority 
of stakeholders. 
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Background 

Collection of individual patient data on Case Report 
Forms (CRFs) in clinical research has traditionally been 
done by investigators in their offices summarizing med- 
ical charts on paper forms (pCRFs), a tedious method 
that could result in data errors and wrong conclusions 
[1,2]. Electronic data capture has in recent years been in- 
creasingly used in both industry and academic research 
settings [3,4]. The feasibility of electronic CRFs (eCRFs) 
has been documented by numerous studies analyzing 



* Correspondence: corinne.alberti@inserm.fr 

2 AP-HP, Hopital Robert Debre, Unite d'Epidemiologie clinique, Groupe 
Hospitalier Robert Debre, 48, Bid Serurier, F-75019 Paris, France 
3 Universite Paris Diderot, PRES Sorbonne Paris Cite, F-75019 Paris, France 
Full list of author information is available at the end of the article 

(3 BioMed Central 



data collected on websites, laptops or digital pens [5-10]. 
Since the mid-1980s, eCRFs have increased data quality 
and completeness by using alarms, automatic completions 
and reminders [11,12], reducing losses and transport logis- 
tics, especially for multicenter trials [13]. Moreover, use of 
eCRFs permits speedier database processing and shorter 
study periods, resulting in lower costs [7,11,14-19]. Previ- 
ous studies of eCRFs have primarily focused on the inves- 
tigators' point of view, while few have documented the 
perspectives of the other stakeholders [5,6,20,21]. 

Despite their demonstrated usefulness, eCRFs have not 
become dominant [3]. Welker has identified some of the 
barriers to their dissemination: the lack of available on- 
site technology, insufficient assistance by Information 
Technologies staff or software providers, investigators' 
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lack of motivation and interference of eCRFs with their 
clinical tasks, complexity of installation and maintenance 
of the software and high investment cost [22]. The labor 
cost of data entry is transferred from clerks to investiga- 
tors who may see few tangible benefits apart from a bet- 
ter quality of data and speedier study completion. While 
some investigators have integrated the electronic inter- 
face into their medical practice to the point of making it 
an asset [23], the majority views the implementation of 
an eCRF in a paper-based working environment as a 
source of redundancy [19,24]. 

Our objective was to formally describe the efficiency 
(measured by satisfaction, cost and duration of the study) 
of electronic and paper CRFs in the context of biomedical 
research conducted in hospitals. 

Methods 

The primary endpoint was the satisfaction of stake- 
holders in a clinical study: investigators, clinical research 
associates (CRAs) and data managers (DMs). Secondary 
endpoints were costs and duration of the studies. Our 
hypotheses were that eCRFs save cost and time [18,22], 
and would be preferred by those 3 stakeholders [5]. 

Material 

Clinical studies: inclusion criteria 

We retrospectively selected biomedical research studies 
monitored by 6 research units involved in eCRF testing, 
completed between 2007 and 2010 (or 2011 for eCRFs) 
and sponsored by the Paris regional hospital consortium 
AP-HP. The research topics were representative of the 
ongoing publicly funded clinical research. Paper (p) CRF 
studies were defined by the use of a CRF on paper, com- 
pleted with a pen, and data entry by a data clerk. Elec- 
tronic (e) CRF studies used computer data entry by the 
investigator or an assistant, online or offline. Two studies 
which used pCRFs to collect data before entering it in the 
eCRF were analyzed in the eCRF group. 

Stakeholders 

We investigated the satisfaction and preference of three 
stakeholder groups: investigators, clinical research associ- 
ates (CRAs) and data managers (DMs) for both types of 
CRF. The investigators surveyed had included patients in 
the studies selected, belonged to 45 different hospitals and 
were primarily physicians or nurses; the CRAs and DMs 
were working for the Paris regional hospital consortium. 

Methods 
Clinical studies 

We collected protocols, budgets and expense statements, 
CRFs, monitoring reports and other relevant technical do- 
cuments. Studies were characterized by phase, number of 
patients, purpose (therapeutic/ diagnostic/observational), 



geographical level and risk (from level A = minimal, 
e.g. trial involving only additional blood sample collection, 
to D = major risk, e.g. trial of innovative therapies, phase I 
or II trials). We estimated the duration of patients' recruit- 
ment, the time between the opening of the first center and 
the database lock, and between the last visit of the last 
patient and the database lock (see Additional file 1). 

Cost estimation 

The cost of a study was estimated from: labor costs, 
i.e. expenditures for CRAs and DMs salaries during the 
study, and logistical costs, i.e. printing of paper CRFs, 
development of database and interface if done by an ex- 
ternal company, cost of the eCRF, travel costs of CRAs 
and investigators, and randomization software. Since 
2003, following a successful tender by TELEMEDICINE 
Technologies S.A.S., the AP-HP s Department of Clinical 
Research and Development {Direction de la Recherche 
Clinique et du Developpement; DRCD) has contracted 
with TELEMEDICINE for use of the software CleanWEB 
in eCRFs for clinical trials. In 2010, the global cost was 
861. 12€ for a trial with one or two centers and 9,687.60€ 
for three or more centers (1€ = US$1.33). We used the 
2010 contract prices for the cost of each eCRF. Expendi- 
tures were updated to 2010 with the hypothesis that they 
were completed during the median year for studies last- 
ing longer than a year. We estimated both the total cost 
and the cost per patient. 

Stakeholders' satisfaction 

Satisfaction of the three stakeholder groups was measured 
through surveys (see Additional files 2, 3 and 4). Each 
stakeholder received their own questionnaire, consisting 
of demographics, closed-ended questions about satisfac- 
tion, preferences and self-reported usage patterns. The 
last part consisted of two open-ended questions to iden- 
tify additional issues. 

Statistical analysis 
Clinical studies 

The unit of analysis was the study. Studies using pCRFs 
and eCRFs were compared by characteristics (phase, num- 
ber of patients and risk), costs and timeframes. Fishers 
exact test was used for categorical variables and Wilcoxon 
sign-rank test for continuous ones. Due to the skewed 
distribution of costs and the sample size, costs per patient 
of pCRFs and eCRFs were compared using the bias- 
corrected and accelerated bootstrap (BCa) procedure with 
2,000 replications to estimate the confidence interval [25] . 
We used a linear regression model to explain costs per pa- 
tient and time between the opening of the first center and 
the freezing of the database. A log transformation was 
used to analyze costs per patient. The linear regression 
models included terms for data collection method (pCRF/ 
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eCRF), randomization, geographical area (regional/ na- 
tional), level of risk, number of patients, number of cen- 
ters and number of variables, time from the opening of 
the first center to database lock. The regression models 



did not include the phase of the study which was highly 
correlated with patient numbers and would not have 
added extra information. A forward stepwise method was 
used, and the variable "pCRF/eCRF" was forced in the 



Table 1 Characteristics of the studies (n = 27) 







pCRFs n = 16 


eCRFs n = 11 


All n = 27 


Design 


Non-interventional studies 


1 (6%) 


4 (36.4%) 


5 (18.5%) 




Clinical trials 


15 (94%) 


7 (63.6%) 


22 (81 .5%) 


Randomized 




10 (63%) 


7 (64%) 


1 / (63%) 


Multicenter 




10 (63%) 


c\ ( c> on/ \ 

9 (82%) 


1 r\ (~i r\c\/ \ 

1 9 (70%) 


Geographic level 


International/national 


7 (44%) 


8 (73%) 


15 (56%) 




Regional 


9 (56%) 


3 (27%) 


1 2 (44%) 


Purpose* 


Diagnostic 


1 (6%) 


3 (27%) 


4 (15%) 




Observational 


0 (0%) 


o / 1 on/ \ 

2 (18%) 


2 (7%) 




Therapeutic 


15 (94%) 


6 (55%) 


21 (78%) 


Risk level" 


A 


2 (13%) 


4 (36.4%) 


6 (22%) 




B 


4 (25%) 


3 (27%) 


7 (26%) 




C 


3 (19%) 


3 (27%) 


6 (22%) 




D 


7 (44%) 


1 (9%) 


O /"") c\c\/ \ 

8 (30%) 


Clinical trial phase 


1 


1 (7%) 


0 (0%) 


1 (4%) 




2 


4 (27%) 


1 (14%) 


5 (23%) 




3 


7 (47%) 


6 (86%) 


13 (59%) 




4 


3 (20%) 


0 (0%) 


3 (14%) 


Median number of patients included* 




60 


355 


80 






Q1 =27 


Q1 = 78 


Q1 = 50 






Q3 = 141 


Q3 = 700 


Q3 = 500 


Median number of centers 




5 


10 


7 






Q1 = 1 


Q1 = 6 


Q1 = 1 






Q3 = 10 


Q3 = 13 


Q3 = 12 


Median planned duration of study (months) 




24 


27 


25 






Q1 = 16 


Q1 = 18 


Q1 = 16 






Q3 = 40 


Q3 = 36 


Q3 = 36 


Median planned patient follow up (days) 




1 27 


DO 


y i 






Q1 =67 


Q1 =12 


Q1 =30 






Q3 = 365 


Q3 = 112 


Q3 = 213 


Median number of variables in CRF 




1,062 


396 


1011 






Q1 =669 


Q1 = 153 


Q1 =286 






Q3 = 1,1 18 


Q3= 1,567 


Q3 = 1,126 


Median number of variables in database 13 




65,928 


304,929 


76,692 






Q1 =18,764 


Q1 =35,250 


Q1 =20,088 






Q3 = 171,646 


Q3 = 625,865 


Q3 = 304,929 


Median number of full pages in CRF* 




39 


17 


31 






Q1 =28 


Q1 =9 


Q1 =17 






Q3 = 44 


Q3 = 30 


Q3 = 44 



"Significant difference between eCRFs and pCRFs (p < 0.05). 

"From level A = minimal, e.g. trial involving only additional blood sample collection, to D = major risk, e.g. trial of innovative therapies, phase I or II trials. 
^Number of variables in database = number of patients x number of variables in CRF. 
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models. Finally, a logistic model was used to explain 
the choice of pCRF/eCRF. The model included rando- 
mization, geographical area, level of risk of the study, 
number of patients included, number of centers, number 
of variables, duration and number of medical teams 
involved [18]. 

Stakeholders' satisfaction 

The unit of analysis was the respondent. Descriptive sta- 
tistics were performed on the answers of investigators, 
CRAs and DMs. Global satisfaction of stakeholders was 
compared using a Chi square test. Multinomial and basic 
logistic regression models were used to understand satis- 
faction and preferences, with the hypotheses that CRAs, 
DMs and youngest stakeholders should prefer eCRFs; 
models included type of stakeholder, age, sex, level of 
computer proficiency and research experience. 

Answers to the two open-ended questions were classi- 
fied by topic using word clustering for the question about 
key features of an optimal data collection form, and by 
topic clustering for the open remarks. 

All analyses were performed using SAS® (Version 9.3 for 
Windows, SAS Institute, Inc., Cary NC, USA). 

Results 

We included 27 clinical studies, 16 (59%) using pCRFs 
and 11 (41%) eCRFs. 

The main characteristics of the studies are summarized 
in Table 1. Electronic CRFs studies were mostly large mul- 
ticenter, national and phase 3 clinical trials while pCRFs 
studies were trials with few patients and centers. The ma- 
jority of pCRFs were drug trials (15/16; p = 0.036), and 
eCRFs were more often used in trials with a significantly 
higher number of patients (355 vs. 60 patients in pCRFs; 
p = 0.014) and fewer data (17 vs. 39 pages with pCRFs; 
p = 0.027). The number of patients was the only explana- 
tory variable for CRF choice (Table 2). 

Clinical studies 

Time from the opening of the first center to database 
lock tended to be shorter with eCRFs (31.7 months vs. 
39.8 months; p = 0.11). We found no difference in the 
average duration of recruitment (22.4 ±9 months with 
eCRFs vs. 26.5 ±13 months with pCRFs; p = 0.34) nor in 
time from the last visit of the last patient to database 
freeze (8 months with eCRFs vs. 8.8 months with pCRFs; 
p = 0.81). Linear regression found that the use of eCRF 
and the smaller number of centers were associated with 
shorter study durations (Table 3). 

The total average cost of a trial was higher with eCRFs 
(88,222€ ± 47,907) than with pCRFs (58,794€ ± 48,665), 
but the mean cost per patient was lower with eCRFs 
(374€ ± 351 vs. 1,135€ ± 1,234) (Figure 1). The difference 



Table 2 Logistic regression model with data collection 
method as dependant variables (n = 27) 



OR 



IC 



Geographic level 


International/ 
national 


1 






0.14 




Regional 


0.292 


0.056 - 


1.525 




Risk level 


A 


1 






0.28 




B 


0.375 


0.039 - 


3.605 






C 


0.500 


0.049 - 


5.154 






D 


0.071 


0.005 - 


1.059 




Number of patients 
included 




1.004 


1 .000 - 


1.009 


0.04 


Number of centers 




1.004 


0.963 - 


1.047 


0.85 


Number of variables 




0.999 


0.998 - 


1.000 


0.23 


Planned duration of 
study 




0.987 


0.928 - 


1.050 


0.68 


Number of medical 
teams involved 




0.936 


0.369 - 


2.374 


0.89 



The modeled probability is the choice of an eCRF for the study as opposed to 
a pCRF. 



in average cost per patient was 762€ per patient (95% CI 
bootstrap: [270€- 1613€]). Other explanatory variables 
were the design of the study - phase 1 and 2 trials without 
randomization being more expensive - and the number of 
patients included, but not the number of variables col- 
lected (Table 3). 



Table 3 Linear regression model with cost log and 
duration as dependant variables (n = 27) 

Parameter SE P value 
estimate 

Duration of the 
study: 

CRF* 



Paper 
Electronic 



Participating 
centers* 

Cost of the study 
(log): 

CRF 



Design* 



-10.14 
0.48 



0.045 
4.79 

0.13 0.001 



Paper 

Electronic 

Trial without 
randomization 

Trial with 
randomization 



0.41 



0.29 



-1.33 



0.35 



Number of 
patients included* 



Non-interventional -2.51 
study 

-0.001 



0.002 

0.40 
0.62 

0,0004 0.021 



SE: standard error. 
^Explanatory variable (p < 0.05). 
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pCRF 





o 








1 



OH ° 



0€ 50 000 € 100 000 € 150 000 € 200 000 € 0€ 1 000 € 2 000€ 

Figure 1 Cost of the studies by data collection method. A: Total cost; B: Total cost per patient. 



Stakeholders' satisfaction 

Thirty-four questionnaires were returned from investiga- 
tors, including six from centers outside Paris. Forty-one 
questionnaires were returned from CRAs and 17 from 
DMs. Seven investigators declined to participate because 
of a lack of experience with eCRFs, and 2 CRAs because 
of a lack of time (Figure 2). Among the 34 answering in- 
vestigators, 33 had experienced pCRFs, 29 eCRFs and 28 
had experienced both and responded to comparison 
questions. The main characteristics of the respondents 
are summarized in Table 4. 

Overall, stakeholders were as satisfied with eCRFs as 
with pCRFs (51/76 vs. 58/86; p = 0.96) (Figure 3). When 
asked for their preference of one over the other, a major- 
ity of stakeholders chose eCRF (Figure 4). 

Half of investigators (16/29) adapted to the eCRF user 
interface from the first patient. Half of them (15/29) expe- 
rienced technical problems from time to time with eCRFs, 
and those were usually resolved within a day (23/29). 
One-third (10/29) was upset by the presence of immediate 
checks and constraints in the eCRF, while most (23/29) 
enjoyed the lack of constraints in a pCRF. Two- thirds of 



investigators (18/29) were satisfied with the intuitiveness 
of the eCRF interface. Twenty /29 investigators never en- 
tered data in the eCRF during consultations, while 15/33 
did so when using a pCRF. Half (16/33) were satisfied that 
the pCRF did not affect or affected positively the patient- 
doctor relationship, while a majority (17/29) had no opin- 
ion on the impact of the eCRF. Data entry was found eas- 
ier with pCRF (13/28 vs. 11/28). Nonetheless, investigators 
were more satisfied (24/29 vs. 9/33) with the logistics, 
storage and data safety of eCRFs. Finally, most investiga- 
tors (24/29) would accept an eCRF in the future while only 
half (17/33) would use a pCRF. 

Most CRAs preferred the eCRF (Figure 3), and one-third 
would choose a CRF depending on the trial characteristics. 
CRAs reported preference for pCRFs in monocentric trials 
including few patients while they would rather use an 
eCRF for a multicentric trial unless there were few patients 
and variables. Despite their preference for eCRFs, CRAs 
identified the following benefits of pCRFs: the greater ac- 
ceptability by investigators, the tangibility of paper and the 
impetus to spend time in the centers to monitor the data 
collection. They based their preference for eCRFs on the 



500 questionnaires sent 



204 investigators 
45 Centres 



270 CRAs 
lOCRUs 



26 DMs 
lOCRUs 



7 refusals 



V 



1 63 no answers 



2 refusals 



V 



227 no answers 



V 



34 questionnaires 
20 Centres 



41 questionnaires 
9CRUs 



1 7 questionnaires 
7CRUs 



92 questionnaires received 



0 refusals 



9 no answers 



Figure 2 Questionnaires flow-chart. CRA = clinical research associate, DM = data manager, CRU = clinical research unit. 
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Table 4 Characteristics of the respondents to the satisfaction and preference surveys 



Investigators CRAs DMs 



Respondent 




34 


41 


17 


Age 


< 30 


0 (0%) 


1 7 (42%) 


7 (44%) 




30 to 40 


8 (24%) 


19 (46%) 


7 (44%) 




> 40 


26 (76%) 


5 (12%) 


2 (12%) 


Gender 


M 


20 (59%) 


6 (15%) 


10 (59%) 




F 


14 (41%) 


35 (85%) 


7 (41%) 


Computer proficiency level 


Beginner 


1 (3%) 








Average 


19 (56%) 








Good 


14 (41%) 






Clinical research experience 


< 1 year 




4 (10%) 


2 (12%) 




1 to 3 years 




19 (46%) 


7 (41%) 




3 to 5 years 




11 (27%) 


3 (18%) 




> 5 years 




7 (17%) 


5 (29%) 



CRA: clinical research associate, DM: data manager. 



more effective monitoring which allowed them to monitor 
data collection from their offices — for example by receiv- 
ing queries of abnormal entries in real time—, the better 
prevention of errors resulting from it and from automatic 
checks, the easier electronic storage (as opposed to the 
copious paper storage of pCRFs) and a greater efficiency 
in managing drug supplies. 

DMs preferred eCRFs (Figure 3) because fewer queries 
were generated and the database contained fewer errors 
before cleaning. Thus they saved time and allowed faster 
data availability. 

No variable was associated significantly with satisfaction 
or preference. The following trends appeared: women 
(DMs and CRAs) were likely to prefer eCRF, younger and 



computer-proficient investigators were likely to be dissat- 
isfied by eCRF, and stakeholders with a greater pCRF ex- 
perience (>10 clinical studies) were likely to prefer pCRF 
(see Additional file 5 and Additional file 6). 

The requirements of an optimal data collection are 
summarized in Table 5: "eCRFs would be perfect at 100% 
if we could have the computer next to the patient". Add- 
itional issues are summarized in Table 6. 



Discussion 

In this first description of the use of eCRFs and pCRFs 
across 27 clinical studies, we found that most stake- 
holders were satisfied with eCRFs and that the use of 



100 



80 



60 



40 



20 



I 



eCRF pCRF 




eCRF pCRF 




eCRF pCRF 


Investigators 




CRAs 




DMs 



Figure 3 Satisfaction of respondents regarding eCRF and pCRF data collection. Percentage of satisfaction level for the 3 stakeholders 
(very satisfied: dark blue, fairly satisfied: light blue, no opinion: yellow, fairly unsatisfied: light red, very unsatisfied: dark red). CRA = clinical research 
associate, DM = data manager. 
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100 



80 



*r eo 



40 



20 









CRAs 



DMs 



Investigators 

Figure 4 Preferences of respondents between eCRF and pCRF data collection. Percentage of stakeholders preferring pCRF (red), with no or 
mixed opinion (yellow) or preferring eCRF (green). CRA = clinical research associate, DM = data manager. 



eCRFs was associated with shorter study duration and 
lower cost per patient. Average duration of recruitment 
did not differ between pCRF and eCRF studies, despite a 
greater number of trials investigating rare and pediatric 
conditions in the pCRF group. 

Data managers reported that eCRFs saved time and 
improved data quality, however we cannot exclude that 
the perception of fewer queries simply results from a 
smaller number of variables monitored; clinical research 
associates valued the automated quality checks and 
easier storage of electronic data. Both DMs and CRAs 
preferred eCRF for multicenter trials. Improvements are 



needed to facilitate the integration of eCRFs into clinical 
practice, including widespread adoption of portable de- 
vices such as digital pens [6] or graphic tablets, which one 
investigator described as "the solution for the future, and 
already used very successfully in anesthesia ". 

It appeared that stakeholders' characteristics do not pre- 
dict preferences, except for young and computer literate 
investigators who tended to be more demanding with 
eCRFs, probably because they had greater expectations of 
eCRF and yet were more aware of its limitations. 

In addition to exploring stakeholder perspectives, we 
found that eCRFs were mainly used in large, national and 



Table 5 "What are the key features of an optimal data collection method in a clinical study?" 





Main features answered 


Details 


Investigators 


Quality interface (x23) 


Fast, simple, without bugs and blocking, with flexible data entry 




Reliable data collection, with few queries (x8) 


Alarms and mandatory fields 




Electronic format (x3) 


Allowing data sharing without data recapture and duplication 


CRAs 


Quality interface (x24) 


Fast, simple, without bugs and blocking, with flexible data entry 




Accessible form (x7) 






Efficient monitoring (x5) 


Real-time queries and remote consultations 




Quality of form (x5) 


No free answers, short and clear questions and few variables 




Immediate controls during data entry (x3) 






Motivation and availability of investigators (x3) 




DMs 


Reliable data collection, few queries (x17) 


Controls and constraints during data entry and queries emailed automatically 




Quality of database, easy to operate (x5) 






Quality of form (x5) 


Real-time queries and remote consultations 




Quality of form (x5) 


No free answers, short and clear questions and few variables 




Quality interface (x3) 


Ergonomic 




Maximum access fees and free action for data managers (x2) 





Responses from investigators, clinical research associates and data managers. 
CRA: clinical research associate, DM: data manager. 
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Table 6 Main themes discussed by stakeholders in open-ended questions 



Investigators Complaints 



About redundancy of data "It doesn't matter whether it's on paper or electronic, as long as data are 
entered only once." 

That some promoters want them to complete a pCRF first as source document and then to re-enter all 
the data in the eCRF. 

About software design companies "By trying to make money, firms that sell this type of CRF software 
developed templates that do not fit well with the variability of studies and data." 

To have their needs taken into account "eCRFs are developed by those who use data but never by 
those who enter the data and who have, in the present context, less and less availability." 

Working with transportable computers 'The graphic tablet, an eCRF transportable to the bedside, is the 
solution for future. It's already used in anesthesia with great success." 

CTAs needed on site: "Whatever the collection method, investigators don't have the time ..." 

eCRF storage: CRFs still needs to be kept on paper, as source data and to be signed ... 

Complaints about CleanWEB "Only simple checks can be defined in CleanWEB; more complicated ones must be programmed in SAS 
consistency management after database export." 

Moreover, "the computer code managing the automatic controls should be accessible and easily understood." 

'The database structure isn't known when designing eCRFs for CleanWEB, yet it is the first thing that 
must be established. And it's currently impossible to have a structure that complies with CDISC." 

Answers from investigators, clinical research associates and data managers. 
CRA: clinical research associate, DM: data manager. 



CRAs 



DMs 



Hopes 



Complaints 



multicenter trials, whereas pCRFs were used in small high 
risk drug trials, possibly because of the greater reliability 
of written documents. 

The few studies investigating stakeholders' experiences 
of CRFs have mostly focused on investigators and found 
a high level of support for eCRFs [24]. We included two 
additional key stakeholders and revealed more mixed re- 
sults in the level of satisfaction and preferences regarding 
eCRFs. CRAs agreed with the Litchfield hypothesis [5] 
that monitoring would be more efficient with an eCRF. 
Moreover, as expected [17,26], acceptance of eCRFs by 
investigators remained one of their biggest challenges. 
Lopez-Carrero reported very positive reviews of eCRFs by 
investigators, with three-quarters finding data entry easier 
and more than a half saying that it decreased workload. 
We did not find investigators to be so optimistic about 
eCRF capacities, particularly with respect to data entry 
and workload. The fact that the Lopez-Carrero [21] 
investigators were asked about a pilot project with a 
"specifically designed" eCRF may explain our less enthusi- 
astic responses. 

While literature reviews have found favorable results 
for eCRFs in terms of study duration and costs, we 
found that the lower cost per patient was explained by 
the large patient number in eCRF trials. This may be at- 
tributable to our sample which did not include pCRF 
studies with large patient numbers, or eCRF studies with 
small patient numbers and/or to the fact that most of 
the other cost studies were based on models [18] or used 
ad-hoc prototypes [27,28]. 

The retrospective non randomized design of our study 
was dictated by two major issues: time constraints and 



acceptability. Randomization would not have been accept- 
able to investigators who want to be able to make their 
own choices and it also would not have been able to show 
that eCRF and pCRF have their specific indications and are 
not therefore perfect substitutes. There is a kind of Indica- 
tion bias' as shown both in Table 1 and in the stakeholders' 
responses. Indeed we have shown that for some studies 
pCRF is cheaper and appears more trustworthy. In 
addition, unobserved factors must also influence the choice 
between eCRF and pCRF as can be assumed by, for ex- 
ample, the fact that the large difference in the number of 
variables between the 2 collection methods was not corre- 
lated to any of the characteristics of the studies that we 
investigated. 

Whilst the CRA and Investigator questionnaires re- 
sponse rates were low, due to many stakeholders having 
changed institutions and no longer being locatable, they 
emanated from 9/10 AP-HP units and 20 different hos- 
pitals spread over the French territory. Our small sample 
of respondents is mirrored in other studies, Lopez-Carrero 
et al. [21] had a similar problem with only 27 investigators 
answering from 33 centers, and Lium et al. [24] inter- 
viewed 18 physicians, but from only 2 centers. 

Most clinical trials sponsored by the AP-HP currently 
use CleanWEB for eCRFs. As a result, the stakeholders' 
answers were related to that software. Our results may 
not adequately reflect the current situation of eCRF 
since half of the studies using eCRFs had started be- 
tween 2004 and 2006 and are not representative of the 
current capacities of electronic data capture. Recent ad- 
aptations of CleanWEB include certification with Clin- 
ical Data Interchange Standards Consortium CDISC a , 
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the possibility to connect to the server on-line or enter 
data off-line and direct data export to SAS, which may 
improve future satisfaction with eCRFs. 

We were unable to suggest a financial advantage in using 
eCRFs in trials with fewer than 50 patients, given that the 
smallest eCRF trial had 47 patients. Nonetheless, Clean- 
WEB price discounts for monocentric eCRFs could make 
it cost-efficient, even for small trials. Likewise, we were 
not able to draw any conclusions regarding pCRF trials 
with more than 700 patients. We did not take into ac- 
count the annual maintenance package of 125,580€, billed 
to the institution for all studies using CleanWEB. We used 
bootstrap replications for the cost analysis because of the 
small number of studies and because calculation of means 
after log transformation resulted in a comparison of geo- 
metric mean costs instead of arithmetic means. The boot- 
strap method requires the true distribution of the data to 
be adequately represented by its empirical distribution 
[25]. Finally, as we did not select the studies based on their 
characteristics, we had much heterogeneity in terms of 
risk level and trial phase. 

An important aspect in comparing pCRFs and eCRFs 
that we have only touched upon in our questionnaires is 
the management of the delivery of drugs and placebo in 
pharmaceutical trials. Computerized treatment manage- 
ment programs that may be included in eCRF software 
enable investigators to streamline complex processes, in- 
cluding: managing the supply of medication to the cen- 
ters even for long-term treatments, managing intricate 
protocols and overseeing product expirations. This sys- 
tem manages stock control and prevents waste, thus pre- 
venting breaks in inclusions due to supply ruptures and 
avoiding mistakes in treatment delivery to the patient. 

Conclusion 

This study examined eCRFs and pCRFs from the view- 
point of investigators and other important stakeholders. 
It found that eCRFs and pCRFs are used in studies with 
different patient numbers, center numbers and risk. The 
first ones are more advantageous in large, low-risk stud- 
ies and gain support from a majority of stakeholders. 
Our findings also suggest that eCRF and pCRF may not 
be substitutes but complement each other with their 
own specific indications. The choice between paper and 
electronic CRF is a significant step in the design and 
execution of clinical studies; it should be discussed with 
the involved stakeholders and based on efficiency. 

Endnote 

a CDISC: standards of acquisition, exchange, submission 
and archive of clinical research data that enable informa- 
tion system interoperability to improve medical research. 
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